Repository navigation
du: match --exclude against trailing path components - #16699
costajohnt wants to merge 4 commits into
Conversation
Exclude patterns were only tried against the full path and the base name, so --exclude=target/debug missed ./target/debug when the operand was . or ./target. Also try every part of the path that starts after a separator, for operands and traversed entries alike, matching GNU du. Fixes uutils#16691
An operand like a// was kept as is, so entries showed as a//x, unlike GNU's a/x, and the exclude suffix walk could match /x against them. Collapse trailing separators to one before traversal, keeping / and // as GNU does.
|
thanks for the patch! |
| pattern.matches(&path) | ||
| || path | ||
| .char_indices() | ||
| .any(|(i, c)| std::path::is_separator(c) && pattern.matches(&path[i + 1..])) |
There was a problem hiding this comment.
this is now one glob match per component for every entry and every pattern, any perf impact on a big tree?
could we only check the full path and the basename when the pattern has no /?
| if end == bytes.len() { | ||
| return path; | ||
| } | ||
| uucore::os_str_from_bytes(&bytes[..end]).map_or_else(|_| path.clone(), PathBuf::from) |
There was a problem hiding this comment.
a prefix of valid bytes can't fail here, do we need the path.clone() fallback?
A pattern starting with * already matched the whole path, and a pattern without wildcards or separators can only match the last component. Convert the path once per entry instead of once per pattern, and drop the unreachable fallback in collapse_trailing_separators.
|
Thanks for the review! pre-commit.ci: that run never got to the hooks, it stopped at "mergeable check: timeout waiting for merge ref". Running the hooks locally on the changed files passes (cspell, fmt, clippy -D warnings on uu_du), so a re-run should be green. perf: you were right, it was measurable. On a generated tree with 200k files (paths about 7 components deep),
I didn't go with "full path and basename when the pattern has no Instead the scan is skipped only where it can't change the result:
I compared the old and new binaries against GNU du 9.11 with du.rs:873: right, the bytes came from |
|
Binary size comparison: |
Merging this PR will degrade performance by 5.19%1 warning
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
|
GNU testsuite comparison: |
--excludewith a pattern that has a/in it only matched the whole path or the basename, sodu --exclude=target/debug .still counted./target/debug. GNU matches a pattern against the full path and against every trailing part that starts after a/, sotarget/debugmatches./target/debug,x/target/debugand/abs/target/debug, whileget/debugdoesn't. du now does the same through oneis_excluded(pattern, path)used at all three places that check excludes (the operands,safe_duanddu_regular). That also removes aninto_string().unwrap()on the basename that could panic on a non-UTF-8 name.GNU also strips trailing slashes from an operand before walking it (
du .//prints./deep/target, and/,///and////all print as/while//stays//). uutils kept them, so child paths read.//deep/target, and with suffix matching a/-anchored pattern like--exclude=/target .//would have matched where GNU doesn't. Operands from the command line and--files0-fromnow have trailing separators collapsed to one before the walk and before-Dhandling, so a symlink given asla//is still followed.Tests:
test_du_exclude_trailing_componentsruns--exclude=a/bagainst.,aand./aand checksxa/bis kept;test_du_operand_trailing_slashescoversa/,a//,a///and excludes on those operands. Both fail on main. I compared output with GNU du 9.7 (LC_ALL=C) over 34 patterns, 20 operands (including.//,target//, symlinks with trailing slashes,..//w//and/tmp//dir/w) and none,-L,-a,-Dand-H: all 3,400 runs print the same as GNU, where main differed in 1,527.--files0-fromwith trailing-slash entries matches too.Fixes #16691